Veeva Vault to File System | Implementation Template

Considerations

To get this template to run, there are several preconditions to consider. Failing to do so can lead to connectivity errors. These preconditions are:

• Veeva Vault Credentials – Require Vault URL, Username, Password, Vault API Version, Client Id, Vault UserId.
• Veeva Vault FTP Credentials – Require Host, Port, Username, Password.
• JKS Certificate for FTPS connector – Require .jks file for FTPS connector to establish TLS connection with Vault FTP server. Following are the steps to follow to create .jks file, you can use Java Keytool located under /bin directory. Below is an example:
• Run command from /bin directory. keytool -genkey -keyalg RSA -alias selfsigned -keystore vault-ftp-server.jks -storepass password -validity 360 -keysize 2048
  • This will create vault-ftp-server.jks file at current directory.
  • Copy/Import vault-ftp-server.jks file under project directory (src/main/resources).
• While Deploying on Cloudhub it is recommended to select more than 1 vCore based on documents metadata volume, number of records and its size of each record. For Migration integrations this is especially important otherwise memory error may occur.
• Veeva System has upgraded Vault FTP Staging server to be more scalable, due to changes on Network load balancing FTP client (Java FTP Client) need to send flag (ftpsClient.setRemoteVerificationEnabled to false). MuleSoft FTPS Connector (v1.4.1) by default set (implicitly) the above flag as true and there is no option available in connector configuration to set the above flag. An enhancement request is logged with MuleSoft engineering. With the above changes/upgrades the FTPS connector fails to connect with Vault FTP server currently and to overcome this issue instead of using the hostname as the Vault DNS, use the POD/FTP specific DNS, you have to check with Veeva Support to provide POD/FTP DNS.
• The FTPS Connector (v1.4.1) is not reusing connection from pool after operation, this ends up in exhausting max connection (50) on Vault FTP Staging server and hit into an error max connection exceeded. A defect is logged with MuleSoft engineering to address this issue. As a workaround you can configure Pooling profile in FTPS Connector configuration in advanced tab. For e.g:
  • Max Active: 5
  • Max Idle: 5
  • Initialisation Policy: INITIALISE_ONE
  • Exhausted Action: WHEN_EXHAUSTED_GROW
  • Max wait: 5
  • Min Eviction: 5000
  • Eviction Check Interval: 5000

Run Stand Alone

In this section we detail the way you should run your template on your computer.

Where to Download Anypoint Studio and the Mule Runtime

If you are new to Mule, download this software:

• Download Anypoint Studio
• Download Mule runtime

Import Template in Studio

In Studio, click the Exchange X icon in the upper left of the taskbar, log in with your Anypoint Platform credentials, search for the template, and click Open.

Run in Studio

After opening your template in Anypoint Studio, follow these steps to run it:

Retrieve Documents Detail from Vault.

1. Locate the properties file configuration.yaml in src/main/resources/config.
2. Complete Veeva Vault and Vault FTP server credentials in Properties to Configuresection.
3. Right click the template project folder.
4. Hover your mouse over Run as.
5. Click Mule Application.
6. Click Run.
7. Once deployment successful, trigger the flow (retrieve-documents-detail) through browser using URL E.g – https://localhost:8082/get-documents?WhereClause=id CONTAINS ('259795', '259794')=true to retrieve documents detail.
8. You will see export documents job status retrieved on browser in JSON format.
9. Verify the JSON file (get-documents-response.json) created at location “${mule.home}/apps/${app.name}/” contains documents detail. E.g – In AnyPoint Studio it will be at (/plugins/org.mule.tooling.server.4.x.x.ee_X.X.X/mule/apps//data/output).
10. The VM Listener flow (files-download-from-vault-ftp) downloads document files exported on Vault FTP Staging location (e.g: /uXXXXXXX/TestDocument.docx) where :

     1) uXXXXXXX - System userId of user defined in connector configuration
     2) JobId - JobId created while export operation is executed.
     3) documentId - Document Id folder of files to be downloaded.
     4) Major-minor-version - Major and Minor version of document file.

1. The downloaded file is saved on local filesystem at location ( /${mule.home}/apps/${app.name}/files/uXXXXXXX/TestDocument.docx) folders structure.
2. Verify document files are downloaded at (${mule.home}/apps/${app.name}/files/uXXXXXXX).

Retrieve Document Renditions Detail from Vault.

1. Right click the template project folder.
2. Hover your mouse over Run as.
3. Click Mule Application.
4. Click Run.
5. Once deployment successful, trigger the flow (retrieve-document-renditions-details) through browser using URL E.g – https://localhost:8082/get-document-renditions?WhereClause=id CONTAINS ('259795', '259794')=true to retrieve document renditions detail.
6. You will see export documents job status retrieved on browser in JSON format.
7. Verify the JSON file (get-document-renditions-response.json) created at location “${mule.home}/apps/${app.name}/” contains documents detail. E.g – In AnyPoint Studio it will be at (/plugins/org.mule.tooling.server.4.x.x.ee_X.X.X/mule/apps//data/output).
8. The VM Listener flow (files-download-from-vault-ftp) downloads document and renditions files exported on Vault FTP Staging area location (e.g: /uXXXXXXX////renditions/TestDocument.docx) where :

     1) uXXXXXXX - System userId of user defined in connector configuration
     2) JobId - JobId created while export operation is executed.
     3) documentId - Document Id folder of files to be downloaded.
     4) Major-minor-version - Major and Minor version of document file.

1. The downloaded file is saved on local filesystem at location (${mule.home}/apps/${app.name}/files/uXXXXXXX////renditions/DocumentRenditionsTest.pdf) folders structure.
2. Verify document files are downloaded at (${mule.home}/apps/${app.name}/files/uXXXXXXX////renditions/).

Retrieve Object Records Detail from Vault.

1. Similarly, for the flow - retrieve-object-records.
2. To retrieve Object records detail, trigger the flow through browser using URL – https://localhost:8082/get-object-records? ++WhereClause=id CONTAINS ('0SI000000000101', '0SI000000000102', '0SI000000000103')++.
3. You will see Object records detail display on browser in JSON format.
4. Verify the JSON file (get-object-records-response.json) created at location “${mule.home}/apps/${app.name}/” contains Object records detail. E.g – In AnyPoint Studio it will be at (/plugins/org.mule.tooling.server.4.x.x.ee_X.X.X/mule/apps//data/output).

Run Stand Alone

Complete all properties in one of the property files, for example in configuration.yaml and run your app with the corresponding environment variable to use it.

Run in CloudHub

After adding your application to Runtime Manager, go to Manage Application > Properties to set the environment variables listed in the Properties to Configure section.

Deploy in CloudHub

In Studio, right click your project name in Package Explorer and select Anypoint Platform > Deploy on CloudHub.

Properties to Configure

To use this template, you need to configure properties (credentials, configurations, etc.) either in a properties file or in Runtime Manager as Environment Variables.

Application Properties

https:
  host: "0.0.0.0"
  port: "8082"

veeva:
  vaultURL: your-vault-url.
  username: your-vault-username.
  password: your-vault-password.
  version : vault-api-version (e.g: v19.2).
  clientId: {company}-{organization}-{server}-{component} format.
  jobId: ""
    export:
      batch: "1000"

ftp:
  host: your-vault-ftp-host.
  port: your-vault-ftp-port.
  username: your-vault-username.
  password: your-vault-password.
  reconnection:
    frequency: "60000"
    retry: "10"
  max:
    active: "10"
    idle: "1"
    wait: "60000"
  move:
    path: "/moved"
  eviction:
    time: "5000"
    interval: "5000"
  working:  "/"

vm:
  queue:
    name: "export-document-queue"
    consumer: "1"
    timeout: "60"

flow:
  max:
    concurrency: "4"

file:
  path:
    workingdir: "${app.home}"
    response: "data/output/get-documents-response.json"
    outputGetObjectRecords: 'data/output/get-object-records-response.json'
  polling:
    interval: "240 "
  start:
    delay: "1"